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3. To establish the date of completion of the invention of this 
application, the following attached document is submitted as 
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the invention disclosure submission form. 
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reference. 
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S-STIR: Similarity Search Through Iterative Refinement 



Problem Solved by Thi. Invention (summuj) ~ ' — ■ 

be^' tt t rf *T ■ nuhn * " ba8Cd ° n tte station of the Euclidean distance 

between the target feature vector and the feature vectors in the database. Euclidean distance 

^rrr^r t nec ^ sarily reflect rdative shniiarity bv the ««■ * paper; 

a method based on nonlinear multidimensional scaling is proposed to provide a mechanism for the 
user to dynamically adjust the similarity measure. The results show that a significant improvement 
on the precision versus recall curve has been achieved. 



BACKGROUND INFORMATION 



( e) Keywords for database search for related work: Invention Disclosure 

(a) Critic*! Dates: 

(e) Suggested TEC: Anant Jhingran 

INVENTOR ON INTERNATIONAL ASSIGNMENT: Is any inventor of thi. disclosure in thi, country on assent from another county 
0 YES, □ No. If 'Yes', see Instruction #5. 



Problem solved by this invention 



Recent methods for retrieving images and videos by content from large archives utilize feature descriptors and 
feature comparison metrics in order to index the visual information. Examples of such content-based retrieval 
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environment commonly ansmg in various scientific applications: cnanging 

* wmS^ T^f' ***** (SAR) Satellite iau «- «* ident ^ *tose regions in the images 

with similar ice type (can be recognized through texture features) to the search target, * 

. Retrieve those one-meter resolution satellite images and identify those regions in the images with similar 
crop type (can be recognized through spectral features) to the search target, 

. Retrieve those LANDSAT Thematic Mapper (TM) satellite images and identify those regions in the images 
s"ar"et ^ * ""^ tWgh * ° f ^ feaTurts)TotI 

Consequently, each image in the three scenarios described above usually has applications in more than one domains 

The ^SEEk P^ject at Columbia University [8] and 
■SLrfnTv^ Pr ° jec * J 6 ' ? **ve developed linear transformations of texture feature spaces. Image database 

££££T7 f ^ *° bCen preVi °^ for ~ * - Pi <*^ [3]. m pfcHunt the 

STS^tS? f T t0 Tk™' 8y8tem ' 8 6Stimate ° f thC g0al ^ A aUto. learning Wed 
on probabilistic model of the user behavior is combined with user selection to estimate the probability of each 
mage in the database. All of these methods definitely provide some improvement in image i^TpS^? 
However, these systems and their use of fixed transforms do not model the user and retrieval 

reS^TrL 8 7 on -wing conditions, context, and the corresponding 

retrieval task. The feature comparison should adapt to these variables. 
Description of this invention 

SJ^J^TS**™' E ?l 6Cl08e 3l ? 0ritlmi *° thC reVi8ion ° f featUre •»* metric formations based upon 

ca^ MH^ ^ ! retrieval P™* 88 - This algorithm is based upon nonlinear multidimensional 

Iv!Z Vf , • ! re ^ 68 ! f f ^ 8paCe ba8ed Up0n the nser ' 8 ^nation of the retrieval results. In this 
system the linear transform of the features is modified by the user's feedback. Furthermore, a deepest gradient 

t^^^^T "* * * — * ^ — * ?W *J^f to the 
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A brief summary is given in Section 5. '°° 4 tt «™Pl«>>«>tation and experiments 
1 Preliminary 

Euclidean metric. For two n dimLional fX ? ^ * b<58t matches « ret ™ed using a 

Euclidean distance between these v^tot is aX7 M ^ " = K ^ - d V = ^ <T. Q» 



*(«, v) = [ (u - vf(u - v)]>" = ^|>-t,,)> (l) 
or, in general, the V distance metric which is defined as 

/ n \ 1/J> 

!>,(«. v) = (jCk-tf.l'J , v>e[i,oo), 

= max)^ - ^1, for p = oo. 

^/in^„n W t^L~™ ^ <W 10 *"»* *— ~ *« «o tne « for 

We performance of tie retrieval is measured in terms of precision and recall, denned below let X be a tamriat, 
n, b. tie number of matches in tke database, «, be tbe requested number of results. Cni»jn?^ 
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" R?Xn)- vf* 6 W 6 .T POrt \ 0 \° f rdeVant reSUltS that *" retrieved - F °' ««* template X let 
**(X,n) - min(n x ,n), the proportion of correct results in a retrieved set of size n. ThenV 

*a(«q) = E*UU*, n Q )} = Ex [ N c(X,n Q ) J 

L mm (n x ,n Q ) J 

2^^^ r^Jlr e ?T ted * by Sam ^ e - d P~c«k» versus recall plots 

lor eacii template X are obtained by varying n outside the range [0, n x \ where R B = R A . 

2 PROPOSED METHOD 

aLoSlTtd tT Z tUne ^ multid ^^al scaling technique that is used in the proposed 

algorithm, and then outline the iterative refinement procedure. proposed 



2.1 Nonlinear Multidimensional Scaling 



^i^^t^' Wfi,,eiM,lt PT ° CeSS ^ t0 diSC ° Ver ^ b6St tra ^ ormati °» ««* that the set of vectors in 

m WlUle ^ dlStanCe b6tWeen ^vectors in different classes is preserved 

This method is based on the multidimensional scaling method proposed by Webb [9]. The objective is to discover 
formation to transform all of the x/s in an n-dimensional vector space X to ^ in an i^ltZZ 

y,- = Wflx,) (4) 

such that the following cost function 

J = (1 - X)J„ + XJ, p ^ 

wh^^' Y f ? Cti ° n ' J J 'l, V da8S J, p is a structure preserving criterion, 

where W 18 an I x m matrix, and is a radial basis function where the i<» component « = 1, ... /) is defined as 



*(x) = exp (-fe^Sl) 
The parameter h? is a bandwidth term where larger value gives rise to a smaller bandwidth. 
The class separability sums up the intraclass distance for all the pairs of vectors defined as below: 



(6) 



(7) 
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where a, and Uj are the class labels of vector x, and x„ and 

Hi = If(x.) - f(x,)| = |W(tf*) - ^(x,.))|, 
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otii's are positive weights, defined as 



{ 0,/or U)i^u) h 



(8) 



(9) 



where ^(X) is the .Euclidean distance between x, and x,: 

^•(X) = |x,-x,| 
The structure preserving criterion is defined as below 

» 3 



(10) 



(11) 



(12) 



I. ha, w !k o m [9, «U. tt e optu^. ^tion W to Eq. 5 is *. ^ to tte 



where 



and 



AW = D(V)V 
A = EE"* ~ + A]fa, - ^-)(& - 

D ( v ) = EE^(v)^-^)(^-^.)* 

» i 



.. (V ) = I *i*i(X)/fc/(V) 6 S + 



(13) 
(14) 

(15) 
(16) 
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2.2 S-STIR: Similarity Search Through Iterative ReBnement 

dX'Tm«rat d i ? ati,re "2*"°* UM ^ £ ' ° f "* * ™ W » «« most .Mar to the 

If the set of the feature vectors selected by the user up to step (t - 1) is denoted as JT,_ tf then 

-X* = -JTt-i U Ui ^27j 
where U t is the set of feature vectors selected during step i. 
The vectors that are NOT selected up to step t - 1 is Y^ Xt then 

Y^Y^UV* (lg) 
where V. is the set of feature vectors rejected during step t. 

hl^oUows^* 11111 ^ Pr ° P08ed alg0ritlmi f ° r it6ratiVe refinemeilt th ™& multidimensional scaling 

L 5lS-f?f ty 86ard T a featUTe VeCt ° r ' V * retrieving the K most similar - «* feature space. 

The similarity between v and u is measured by Eq. 1. Set i = 1. 

2. Initialize and U, to those vectors which are consider to be similar. Also initialize Yi and K to those 
vectors that are not considered to be similar. If the number of vectors is less than a prescribed Jhreshold, 
K ine , where K inc is a fixed increment. Return to step 1. 

3 ' ve^orT IfZSfT*"? &C ff d r? ed earUer iD the preVioUS Subsection based « of 
l^Zl /I ' W , , ' ^ ^ VeCt ° rS that " e consider ^ similar, while y 4 include all the 

vectors that are considered to be no similar (or irrelevant). Consequently, the class label for X- is 1 while 
the class label for Y t is 2. The resulting W optimize the objective cost function /, as defined fa Eq 5 

~~ — n — _ -1 i U I \/w f-,,- 




7 = IBM Confidential 

Invention Disclosure 
Page 7 of 16 
Disclosure number 

4. Perform similarity search in the scaled feature space. The similarity measure beta**, t™ ♦ • v 
space is measured by \W*(6(u) - tk» «L.h "^"yy measure between two vectors m the new 

(or irrelevant). Assuming that V^V^^T^^ ( °' *** **** 

respectively. * ' * ^ 8CtS that mdude tW m* dissimilar vectors, 

5. Update Jf, and Y< as follows 

** = *i-iUDi. (lg) 

(20) 

thejfference between X< and X,., is less than a prescribed threshoid, a equilibrium has been reached 

6. set t = t + 1, and return to step 3. 

Note that there are a number of possible strategies to handle relevance feedWV Tfc„ v a -i. * . . 

3 IMPLEMENTATION AND EXPERIMENTS 

The feature vector used in the database has 21 dimensions, consisting of spatial texture features such as fractal 
pension coocurrence-based texture features, spatial gray level diLenclbased ^ZTZ ZlTs 
skew, dispersion and Moran circular correlation. The feature database is generated as follows: We ge^ aTe 3 2 
random cuts of size 32x32 from each of the 37 satellite images, each of which consist of homogenius^eTSns 

total of 1184 feature vectors For each query, one of the random cuts from an image is used to retrieve the K 
most similar random cuts. The retrieved result is considered to be a hit if the the retrieved random cut belongs to 
the same image as the original random cut. Note that the precision and recall values given in this section are all 
average values usmg Eq. 2 and Eq. 3. Figures 3 and 3 show examples of mountain, woods, forests, and suburban 
areas used in the 37 benchmark images. 

To test the algorithm, we retrieve the first K (K varies from 64 to 256) feature vectors as samples and assign class 
labels. Note that only two feature classes will be covered if K is equal to 64. The iterative refinement algorithm 
outlined in the previous section is then applied to the retrieved feature vectors together with its feature class The 
resulting W is applied in conjunction with the radial basis function defined in the previous section to transform all 
the feature vectors in the database. A nearest neighbor search is then applied tb'dctvrmine tie resulting precision 
and recall values. • * . ?r .. 
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Figure 1: Satellite images of (a) mountain (b) mixed area. 



Figure 3 shows the precesion versus recall for a given benchmark image before and after the S-STIR algorithm is 
applied. In this case, the sample size is selected to be 256, the number of radial basis functions is chosen to be 
-0 (thus requiring the clustering function TSVQ to generate 20 clusters from 256 vectors), and the final feature 
Z°; * P *£ T 10 The Parameter h is set at 3.16 throughout the experiment [9]. It is quite apparent 

tnat i>-bllK algorithm produced a significant improvement on the precision for a given recall, and vice versa The 
improvement is bigger for larger recall (or smaller precision). 

To observe the impact from the parameter A, we vary X from 0.2 to 0.8. Smaller A implies less emphasis on J,„ 
and more emphasis on J, e , resulting in better class separability (See Eq. 5). This is evident from Fig. 4, as lower 
A results in better precision versus recall curve. 

Larger initial sample size is important to the S-STIR algorithm, as it requires a better global view of the entire 
database to determine the transformation. Figure 5 shows that the precision versus recall performance dramatically 
deteriorates as the sample size is reduced from 256 to 64. Sampling techniques of the space can be applied and 
may produce a better training set for the S-STIR algorithm. This, however, is beyond the scope of this paper. 
The number of radial basis function, and thus the number of clusters needed to be generated from the TSVQ 
clustering algorithm, also has an impact on the performance. As shown in Fig. 6, a point is reached when 
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Figure 2: Satellite images of (a) forests (b) suburbs. 



* agmented the feature space -* do not ^ to * 

The additional bonus of using the nonlinear multi-dimensional scaling technique is the reduction of dimensionality 
a AS , Shoy ? 1 m 7 ' * he number of dunensions that are required for clean separation between the desirable 
and undesirable results is less than 5. Consequently, the precision versus recall curve are space fairly closely with 
respect to each other when the final number of dimensions is varied from 5 to 15. 



4 SUMMARY 



Sirnilarity measure has been one of the critical issues for successful content-based retrieval. Simple quadratic forms 
of distance is inadequate as it does not necessary correspond to perceived similarity nor is it adaptive to different 
applications. 

In this paper, we propose an iterative refinement algorithm for Content-based retrieval of images based on low-level 
features such as textures, color histogrammed shapes that can be describe by^ature v&tors. This technique 

-P3- 
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Figure 3: Comparison of precision versus recall for raw feature vector and feature vectors transformed by S-STIR 
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Figure 4: Effect on the precision versus recall for different values of A. 



WITNESSES: The two witnesses whose signa- 
tures appear below have reed and understand 
this entire invention disclosure 

Signature of Witnesses Date" 


DISCLOSURE SUBMITTED B^fc- N/ 




xnvwtor^S^nature ^ Date 


Inventor's Signature T?? ^ n . tj> 




inventor!* Signature * " — Sato"" 








Inventor's Signature^ — Date" 




Inventor 1 * Signatuf*^ " — Dfctfl 



IBM Confidential 



Invention Disclosure 
Page 12 of 16 

Disclosure number 



0.6 



0.55 



0.S 



I 0.46 

o 

111 

CC 

o- 0.4 



0.3S 



0.3 



SAMPLE SI2E-256 
SAMPLE SIZE-64 



RECALL 



0.25}- ■ 1 — 1 1 . __, , 



Figure 5: Effect on the precision versus recall for different values of sampL 
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Figure 6: Effect on the precision versus recall for different number of clusters. 
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RECALL 

Figure 7: Effect on the precision versus recall for different values of final dimensionality. 
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new feature space. Meanwhile, the stance „ S^SZf™* ,0 "* ^ fc «• *e 

-»•<»*-. space each time a ns£ make JES^^"?* ^ * ^ 

featnr, vector, from a large datable becomes a serious Lne agata^ Consequently, efficient retrieval 
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Alternatives 

Potential Use 

Potential applications include: 



1. Environmental epidemiology: retrieve locations of houses which are vulnerable to epidemic diseases such as 
Hantavirus and Denge fever based on a combination of environmental factors (e.g. isolated houses that are 
near bushes or wetlands), and weather patterns (e.g. a wet summer followed by a dry summer). 

2. Precision farming: retrieve locations of cauliflower crop developments that are exposed to clubroot, which is 
a soil-borne disease that infects cauliflower crop. Cauliflower and clubroot are recognized spectral signature 
and exposure results from their spatial and temporal proximity. ' 

3. Medical image diagnosis: retrieve all MRI images of brains that have tumors located within the hypothala- 
mus. The tumors are characterized by shape and texture, and the hypothalamus is characterized by shape 
and spatial location within the brain. 

4. Real estate marketing: retrieve all houses that are near a lake (color and texture), have a wooded yard 
(texture) and are within 100 miles of skiing (mountains are also given by texture). 

5. Interior design: retrieve all images of patterned carpets which consist of a specific spatial arrangement of 
color and texture primitives. 
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